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1.  Introduction 


1.1  Purpose 

The  goal  of  this  research  was  to  examine  the  effects  of  unreliable  alerts  on  gunners’  concurrent 
performance  of  gunnery,  robotics,  and  communication  tasks.  We  simulated  a  generic  mounted 
(i.e.,  tank)  crew  station  environment  and  incorporated  tactile  and  visual  displays  to  provide 
directional  cueing  for  the  gunnery  targeting  task  (based  on  simulated  aided  target  recognition 
[AiTR]  capability).  Two  types  of  imperfect  AiTR  were  simulated:  false-alann  prone  (FAP)  and 
miss-prone  (MP).  Effects  of  individual  differences  such  as  spatial  ability  (SpA)  and  perceived 
attentional  control  (PAC)  were  also  evaluated.  We  were  also  interested  in  investigating  discrepan¬ 
cies  in  previous  research  related  to  compliance  and  reliance  effects  as  a  function  of  type  of  AiTR 
error. 

1.2  Background 

The  current  U.S.  Army  Future  Combat  Systems  (FCS)  concept  for  the  Mounted  Combat  System 
(MCS)  is  that  it  will  be  operated  with  a  crew  of  three  Soldiers:  a  vehicle  commander,  driver,  and 
gunner;  however,  there  is  no  dedicated  operator  to  control  the  platoon’s  unmanned  ground  vehicle 
(UGV)  (TRADOC,  2003).  To  examine  how  the  MCS  crew  can  effectively  employ  its  UGV  and 
which  crew  member  should  serve  the  role  as  the  robotics  operator,  Mitchell  (2005)  conducted  a 
workload  analysis,  using  the  Improved  Perfonnance  Research  Integration  Tool  (IMPRINT)  to 
model  the  workload  of  each  crew  member  if  s/he  has  to  concurrently  operate  a  UGV.  Mitchell 
found  that  the  gunner  had  the  fewest  instances  of  overload  and  could  assume  control  of  the  UGV 
and  its  associated  tasks.  However,  there  were  instances  in  the  model  when  the  gunner  dropped  his 
or  her  primary  tasks  of  detecting  and  engaging  targets  to  perform  robotics  control  tasks,  which 
could  be  catastrophic  for  the  team  and  mission  during  a  real  operation.  Additionally,  in  scenarios 
when  the  UGV  required  tele-operation,  the  gunner  was  found  to  be  consistently  in  overload,  mak¬ 
ing  concurrent  performance  of  other  tasks  even  more  challenging. 

In  order  to  further  investigate  issues  derived  from  Mitchell’s  IMPRINT  analysis  and  to  verify  its 
results,  we  conducted  a  series  of  three  simulation  experiments  to  examine  the  mental  workload  and 
performance  of  the  combined  position  of  robotics  operator  and  gunner.  In  the  first  study,  we  found 
that  when  the  robotics  operator  must  perform  robot  targeting  and  local  security  (i.e.,  gunner’s 
tasks)  at  the  same  time,  both  workload  and  performance  degraded  (Chen  &  Joyner,  2006).  Our 
results  showed  that  a  gunner’s  target  detection  performance  degraded  significantly  when  partici¬ 
pants  had  to  concurrently  monitor,  manage,  and  tele-operate  a  UGV,  compared  with  the  baseline 
condition  when  they  only  had  to  perfonn  the  gunnery  task.  As  the  robotics  task  became  more 
challenging,  participants’  gunnery  performance  degraded  accordingly.  Their  gunnery  performance 
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was  worst  when  they  had  to  tele-operate  a  robot  simultaneously.  For  the  robotics  tasks,  partici¬ 
pants’  performance  was  lowest  when  they  controlled  a  semi-autonomous  UGV  (only  53%  of  the 
targets  were  detected).  Participants’  perceived  workload  was  lowest  in  the  single  task  condition. 
There  were  three  types  of  robotics  tasks:  Monitor,  Auto,  and  Teleop.  As  the  robotics  task  became 
more  difficult,  participants’  workload  increased  accordingly,  with  the  Teleop  condition  being  the 
highest. 

In  the  second  study  (Chen  &  Terrence,  2007),  we  examined  if  and  how  reliable  tactile  cueing, 
which  delivered  AiTR  information  to  aid  the  operator’s  gunnery  task,  enhanced  a  gunner’s  per¬ 
formance  in  a  multi-tasking  environment  similar  to  Chen  and  Joyner’s  (2006).  We  found  that 
participants’  gunnery  task  performance  improved  significantly  when  it  was  assisted  by  AiTR. 

Those  participants  with  higher  SpA  outperfonned  those  with  lower  SpA,  especially  when  there  was 
no  AiTR.  It  was  also  found  that  significantly  fewer  neutral  targets  (which  were  not  cued)  in  the 
gunnery  environment  were  detected  (implying  less  visual  attention  being  devoted  to  the  gunnery 
station)  when  participants  concurrently  tele-operated  a  robotic  asset  or  when  the  gunnery  task  was 
assisted  by  AiTR.  The  most  likely  reason  for  this  was  that  AiTR  cueing  increased  task  switching 
efficiency  between  the  primary  (gunnery)  and  secondary  (robotics)  tasks.  However,  this  improved 
effectiveness  for  the  gunnery  and  robotics  tasks  had  a  deleterious  effect  on  the  uncued  neutral 
targets.  Participants’  concurrent  task  performance  (i.e.,  robotics  and  communication)  improved 
significantly  when  the  gunnery  task  was  assisted  by  AiTR.  It  was  also  found  that  the  overall  per¬ 
formance  gap  (including  gunnery  and  robotics)  between  those  participants  with  higher  and  lower 
SpA  appeared  to  be  narrower  when  the  AiTR  was  available.  A  similar  pattern  was  also  observed 
for  the  PAC  factor.  Finally,  participants’  perceived  workload  was  significantly  influenced  by  the 
type  of  robotics  tasks  and  whether  the  gunnery  task  was  assisted  by  AiTR.  Partici-pants’  perceived 
workload  was  significantly  higher  when  they  tele-operated  a  robotic  asset  and  when  their  gunnery 
task  was  unassisted.  In  a  post-experimental  survey,  65%  of  the  participants  indicated  that  they 
relied  predominantly  on  the  tactile  cues  when  tactile  and  visual  displays  were  available;  only  15% 
said  they  relied  primarily  on  the  visual  cues.  Those  who  preferred  visual  cueing  tended  to  have 
lower  SpA,  and  their  gunnery  and  robotics  task  performance  tended  to  be  inferior. 

1.3  Imperfect  Automation  and  Multi-tasking  Performance 

As  described  previously,  Chen  and  Terrence  (2007)  demonstrated  the  utility  of  reliable  AiTR  for 
enhancing  the  operator’s  automated  and  concurrent  tasks.  However,  since  no  AiTR  systems  can 
have  perfect  reliability,  Chen  and  Terrence  (2007)  can  only  provide  us  an  estimate  of  optimal 
condition  when  the  alert  is  highly  reliable.  In  the  real  world,  cueing  systems  are  often  FAP  or  MP, 
based  on  the  threshold  settings  of  the  alert.  Wickens,  Dixon,  Goh,  and  Hammer  (2005)  showed 
that  the  operator’s  automated  task  (i.e.,  system  failure  monitoring)  performance  degraded  when  the 
false  alarm  (FA)  rate  of  the  alerts  for  the  automated  task  was  high.  In  other  words,  high  FA  rate 
reduced  the  operator’s  compliance  with  automation  (compliance  defined  as  taking  actions  based  on 
the  alerts).  Conversely,  when  the  miss  rate  was  high,  the  concurrent  task  performance  was  affected 
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more  than  the  automated  task  because  the  operator  had  to  allocate  more  visual  attention  to  monitor 
the  automated  task.  In  other  words,  a  high  miss  rate  reduced  the  operator’s  reliance  on  automation 
(reliance  defined  as  failure  to  take  precautious  actions  when  there  is  no  alert).  Similarly,  Dixon 
and  Wickens  (2006)  showed  that  FAs  and  misses  affected  compliance  and  reliance,  respectively, 
and  their  effects  appeared  to  be  relatively  independent  of  each  other.  In  contrast,  Dixon,  Wickens, 
and  McCarley  (2007)  showed  that  FAP  automation  hurt  “performance  more  on  the  automated  task 
than  did  MP  automation,  (e.g.,  the  “cry  wolf’  effect)  and  hurt  perfonnance  (both  speed  and  ac¬ 
curacy)  at  least  as  much  as  MP  automation  on  the  concurrent  task  (p.  570-571).”  FAP  automation 
was  found  to  affect  operator  compliance  and  reliance,  while  MP  automation  affected  only  operator 
reliance.  The  authors  suggested  that  the  FAP  automation  had  a  negative  impact  on  reliance  be¬ 
cause  of  the  operator’s  overall  reduced  trust  in  the  automated  system.  Similarly,  Wickens,  Dixon, 
and  Johnson  (2005)  demonstrated  a  greater  cost  associated  with  FAP  automation  (than  with  MP 
automation),  which  affected  the  automated  and  concurrent  tasks. 

Furthermore,  Wickens  and  Dixon  (2005)  demonstrated  that  when  the  reliability  level  is  below 
about  70%,  operators  often  ignore  the  alerts.  In  their  meta-analytic  study,  Wickens  and  Dixon 
found  that  “a  reliability  of  0.70  was  the  ‘cross-over  point’  below  which  unreliable  automation  was 
worse  than  no  automation  at  all.”  Although  Wickens  and  his  colleagues  have  done  extensive 
research  in  this  area,  their  studies  were  done  in  a  different  environment  (unmanned  aerial  vehicle 
control  display  monitoring),  and  they  did  not  use  tactile  cueing.  The  current  study  was  the  first 
one  to  examine  these  issues  in  the  context  of  combined  roles  of  gunner  and  robotics  operator. 

Since  an  AiTR  cannot  have  a  perfect  reliability  rate  in  foreseeable  real-world  operations,  the  data 
from  this  study  should  provide  useful  information  to  the  design  community  of  future  systems  such 
as  the  FCS,  in  which  AiTR  will  play  an  integral  role  (see  Chen  &  Terrence,  2007,  for  a  detailed 
discussion  of  the  utility  of  tactile  display  for  AiTR  cueing). 

1.4  Individual  Differences 

In  the  current  study,  we  sought  to  investigate  the  effects  of  individual  differences  factors  such  as 
SpA  and  PAC  on  the  operators’  concurrent  performance.  Previous  studies  (Chen  et  al.,  in  press; 
Chen  &  Joyner,  2006)  found  that  SpA  was  correlated  with  robotics  and  gunnery  task  perfonnance. 
Lathan  and  Tracey  (2002)  demonstrated  that  people  with  higher  SpA  performed  better  in  a  tele¬ 
operation  task  through  a  maze.  They  finished  their  tasks  faster  and  had  fewer  errors.  Lathan  and 
Tracey  suggested  that  military  missions  can  benefit  from  selecting  personnel  with  higher  SpA  to 
operate  robotic  devices.  In  addition  to  SpA,  we  examined  the  relationship  between  attentional 
control  and  multi-tasking  performance.  Derryberry  and  Reed  (2002)  showed  that  those  with 
higher  attentional  control  could  switch  their  attention  among  tasks  more  effectively  and  this  was 
partially  confirmed  by  Chen  and  Joyner  (2006)  and  Chen  and  Terrence  (2007).  There  is  some 
evidence  that  attention- switching  flexibility  can  predict  performance  of  such  diverse  tasks  as  flight 
training  and  bus  driving  (Kahneman,  Ben-Ishai,  &  Lotan,  1973).  Several  studies  show  that  there 
are  individual  differences  in  multi-tasking  performance,  and  some  people  are  less  prone  to 
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performance  degradation  during  multi-tasking  conditions  (Rubinstein,  Meyer,  &  Evans,  2001; 
Schumacher  et  ah,  2001). 

1.5  Current  Study 

In  the  current  study,  we  replicated  the  conditions  of  Chen  and  Joyner  (2006)  and  manipulated  the 
reliability  of  the  AiTR  cueing  system  (i.e.,  false  alarm  and  miss  rates).  Based  on  the  data  from 
Wickens,  Dixon,  Goh,  and  Hammer  (2005),  we  expected  that  the  operator’s  gunnery  (automated) 
task  performance  would  degrade  if  the  FA  rate  of  the  AiTR  for  the  gunnery  system  were  high 
because  of  reduced  compliance  with  the  automation.  Conversely,  if  the  cueing  were  MP,  the 
operator’s  robotics  (concurrent)  task  performance  would  be  affected  more  than  the  gunnery  task 
because  of  reduced  reliance  on  the  automation.  More  mental  and  visual  resources  would  be 
devoted  to  checking  the  raw  data  for  the  automated  task,  and  therefore,  the  perfonnance  of  the 
concurrent  task  would  be  degraded.  On  the  other  hand,  there  was  evidence  that  FAP  automation 
was  more  detrimental  to  the  automated  and  concurrent  tasks  than  MP  automation  (Dixon  et  ah, 
2007).  Therefore,  it  is  likely  that  FAP  automation  would  have  a  more  negative  impact  on  the 
overall  performance  than  would  MP  automation.  In  other  words,  there  have  been  conflicting 
results  in  the  literature  regarding  the  independence  of  the  effects  of  FAP  and  MP  automation  on 
operator  compliance  and  reliance.  It  is  possible  that  individual  differences  may  be  responsible  for 
some  of  the  observed  differences  in  the  literature.  Therefore,  we  investigated  the  effects  of 
individual  differences  on  FAP  and  MP  conditions  as  a  possible  explanation  for  the  discrepancies. 
Additionally,  in  Chen  and  Terrence  (2007),  we  observed  that  low  SpA  participants  indicated  that 
they  preferred  visual  cueing  when  visual  and  tactile  cueing  displays  were  available,  whereas  high 
SpA  participants  favored  tactile  cueing  over  visual  cueing.  In  the  current  study,  visual  and  tactile 
cueing  were  presented.  The  relationship  between  SpA  and  preference  of  AiTR  display  modality 
was  examined. 


2.  Method 


2.1  Participants 

A  total  of  24  students  (4  females  and  20  males)  was  recruited  from  the  University  of  Central 
Florida  and  participated  in  the  study.  The  ages  of  the  participants  ranged  from  18  to  34  (mean  [M\ 
=  22.25,  standard  deviation  [SD]  =  4.0).  Participants  were  compensated  $  15/hr  and/or  class  credit 
for  their  participation  in  the  experiment. 
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2.2  Apparatus 

2.2.1  Simulators 

2.2. 1 . 1  Tactile  Control  Unit  (TCU) 

The  experiment  was  conducted  with  a  TCU  developed  by  the  U.S.  Army  Research  Laboratory’s 
(ARL’s)  Robotics  Collaborative  Technology  Alliance  (RCTA)  for  the  robotic  control  tasks.  The 
TCU  is  a  one-person  crew  station  from  which  the  operator  can  control  several  simulated  robotic 
assets  that  can  perfonn  their  tasks  semi-autonomously  or  be  tele-operated  (see  figure  1).  The 
operator  switched  operation  modes  and  display  modes  through  the  use  of  a  19-inch  touch-screen 
display.  A  joystick  was  used  to  manipulate  the  direction  in  which  the  unmanned  vehicles  moved 
when  in  Teleop  mode.  The  UGV  simulated  in  our  study  is  the  experimental  unmanned  vehicle 
developed  by  ARL.  The  simulation  program  used  in  this  study  was  rSAF  (robotic  semi-automated 
forces),  which  is  a  version  of  OneSAF  for  robotics  simulation. 


OCU  with  RSTA  view 

Figure  1.  User  interface  of  RCTA  TCU. 

2. 2. 1.2  Gunnery  Station 

The  gunnery  component  was  implemented  with  an  additional  screen  and  controls  to  simulate  the 
out-the-window  (OTW)  view  and  line-of-sight  (LOS)  fire  capabilities.  The  interface  consisted  of  a 
15-inch  flat  panel  monitor  and  a  joystick  (see  figure  2).  Participants  used  the  joystick  to  rotate  the 
sensors  360  degrees,  zoom  in  and  out,  switch  between  firing  modes,  and  engage  targets.  In  each 
scenario,  there  were  10  hostile  targets  and  10  neutral  targets  (i.e.,  civilians)  scattered  throughout 
the  simulated  environment. 


5 


Figure  2.  TCU  (left)  and  gunnery  station  (gunner’s  OTW  view)  (right). 

2.2. 1 .3  AiTR  Systems  -  Tactile  and  Visual  Alerts 

To  augment  target  detection  in  the  gunnery  component,  visual  and  tactile  alerts  were  used  to  cue 
the  participant  to  the  direction  of  a  target,  as  determined  by  the  AiTR.  Visually,  the  targets  con¬ 
sisted  of  icons  presented  around  the  overhead  view  diagram  of  the  participant  vehicle  in  the  lower 
right  area  of  the  screen  (see  figure  3).  The  target  icon  appeared  in  one  of  eight  possible  locations 
around  the  gunner,  corresponding  to  45-degree  increments  along  a  360-degree  azimuth.  As  the 
gunner  rotated  the  view,  the  turret  portion  of  the  vehicle  diagram  moved  along  the  eight  possible 
orientations  to  allow  the  gunner  to  place  his/her  field  of  view  (FOV)  on  the  cued  target.  Tactually, 
target  positions  relative  to  the  gunner  were  presented  via  eight  electromechanical  transducers 
known  as  “tactors”  (see  figure  4).  More  details  about  the  tactile  display  are  given  in  Chen  and 
Terrence  (2007). 


Figure  3.  Visual  cueing  (red  dot  indicating  the 
direction  of  hostile  target). 
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Figure  4.  Tactile  system. 


2.2.2  Communication  Task  Materials 

The  communication  task  was  administered  concurrently  with  the  experimental  scenarios.  The 
questions  included  simple  military-related  reasoning  tests  and  simple  memory  tests.  The  inclusion 
of  these  cognitive  tasks  was  for  simulating  an  environment  where  the  gunner  was  communicating 
with  fellow  crew  members  in  the  vehicle.  For  the  reasoning  tests,  there  were  questions  such  as  “if 
the  enemy  is  to  our  left,  and  our  UGV  is  to  our  right,  what  direction  is  the  enemy  to  the  UGV?” 
For  the  memory  tests,  the  participants  were  asked  to  repeat  some  short  statements  or  keep  track  of 
three  radio  call  signs  (e.g.,  Bravo  83)  and  they  had  to  report  to  the  experimenter  whether  the  call 
signs  they  heard  were  one  of  those  they  were  keeping  track.  Test  questions  were  delivered  by  a 
synthetic  speech  program,  DECTalk1.  The  questions  were  pre-recorded  by  a  male  speaker  and 
presented  at  the  rate  of  one  question  every  33  seconds. 

2.2.3  Questionnaires  and  Spatial  Tests 

A  demographics  questionnaire  was  administered  at  the  beginning  of  the  training  session  (appen¬ 
dix  A).  A  questionnaire  about  attentional  control  (appendix  B)  (Derryberry  &  Reed,  2002)  was 
used  to  evaluate  participants’  PAC.  The  Attentional  Control  Survey  (ACS)  consists  of  21  items 
and  measures  attention  focus  and  shifting.  The  scale  has  been  shown  to  have  good  internal  relia¬ 
bility  (a  =  .88).  Derryberry  and  Reed  conducted  an  experiment  to  examine  the  relationship  be¬ 
tween  self-reported  (i.e.,  ACS  score)  and  actual  attentional  control.  They  found  that  participants 
with  a  high  ACS  score  could  better  resist  interference  in  a  Stroop-like  spatial  conflict  task.  In  our 
previous  studies  (Chen  &  Joyner,  2006;  Chen  &  Terrence,  2007),  we  observed  a  positive,  although 
some-what  weak,  relationship  between  ACS  score  and  some  multi-tasking  performance  measures. 


1  DEC  talk  is  a  registered  trademark  of  Digital  Equipment  Corporation. 
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The  Cube  Comparison  (Cube  for  short)  and  the  Hidden  Patterns  tests  (Educational  Testing  Service, 
2007a&b)  as  well  as  the  Spatial  Orientation  Test  (Orientation  for  short)  were  used  to  assess  partici¬ 
pants’  SpA.  The  Cube  test  requires  participants  to  compare,  in  3  minutes,  21  pairs  of  six-sided 
cubes  and  determine  if  the  rotated  cubes  are  the  same  or  different.  The  Hidden  Patterns  test 
measures  flexibility  of  closure  and  involves  identifying  specific  patterns  or  shapes  embedded 
within  distracting  information.  The  Orientation  test,  constructed  by  Dr.  Paula  Durlach  of  the  U.S. 
Army  Research  Institute,  is  modeled  after  the  Cardinal  Direction  test  developed  by  Gugerty  and 
his  colleagues  (Gugerty  &  Brooks,  2004)  and  is  a  computerized  test  consisting  of  a  brief  training 
segment  and  32  test  questions.  Accuracy  and  response  time  were  automatically  captured  by  the 
program. 

Participants’  perceived  workload  was  evaluated  with  the  computer-based  version  of  National 
Aeronautics  and  Space  Administration  Task  Load  Index  (NASA  TLX)  questionnaire  (appendix  C) 
(Hart  &  Staveland,  1988).  The  NASA  TLX  is  a  self-reported  questionnaire  of  perceived  demands 
in  six  areas:  mental,  physical,  temporal,  effort  (mental  and  physical),  frustration,  and  performance. 
Participants  were  asked  to  evaluate  their  perceived  workload  level  in  these  areas  on  10-point  scales. 
They  also  assessed  the  contribution  (i.e.,  weight)  of  each  factor  to  the  perceived  workload  by  com¬ 
paring  the  1 5  possible  pairs  of  the  six  factors.  According  to  Noyes  and  Bruneau  (2007),  computer- 
based  NASA  TLX  tends  to  generate  higher  workload  ratings  compared  with  the  traditional  paper- 
based  survey.  However,  since  the  ratings  were  used  to  compare  the  workload  levels  across  the 
experimental  conditions,  the  elevated  ratings  should  not  affect  these  comparisons. 

Linally,  a  usability  questionnaire  about  the  tactile/visual  AiTR  displays  was  constructed  (see 
appendix  D).  Participants  indicated  their  level  of  reliance  on  tactile  and/or  visual  cueing  for  the 
gunnery  task  when  both  types  of  alerts  were  available.  They  also  indicated  their  perceived 
usability  of  the  AiTR  displays.  Participants  were  also  asked  to  evaluate  their  trust  in  the  AiTR 
system  using  a  modified  survey  by  Jian,  Bisantz,  and  Drury  (2000)  (appendix  D,  items  22-33). 

2.3  Experimental  Design 

The  overall  design  of  the  study  is  a  2  x  3  mixed  design.  The  between-subject  variable  is  AiTR 
type  (PAP  versus  MP).  The  within-subject  variable  is  Robotics  Task  type  (Monitor  versus  Auto 
versus  Teleop). 

The  reliability  level  of  the  PAP  and  MP  alerts  was  60%.  The  PAP  condition  consisted  of  10  hits 
(i.e.,  alerts  when  there  were  targets),  eight  LAs  (i.e.,  alerts  when  there  were  no  targets),  no  misses 
(i.e.,  no  alerts  when  there  were  targets),  and  two  correct  rejections  (CRs)  (i.e.,  no  alerts  when  there 
were  no  targets).  The  MP  condition  consisted  of  two  hits,  no  LAs,  eight  misses,  and  10  CRs. 
Additionally,  only  hostile  targets  were  cued,  not  the  neutral  targets.  The  participants  were 
instructed  to  detect  the  neutral  targets  independently. 
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2.4  Procedure 

After  being  briefed  about  the  purpose  of  the  study,  the  tasks  for  the  experiment,  and  any  risks 
involved,  participants  read  and  signed  a  consent  form.  They  then  answered  the  attentional  control 
survey  and  were  administered  the  spatial  ability  tests  (i.e.,  Cube,  Hidden  Patterns,  and  Orientation). 
After  these  tests,  participants  received  training,  which  was  self-paced  and  was  delivered  by  Power¬ 
Point2  slides  showing  the  elements  of  the  TCU,  steps  for  completing  various  tasks,  several  mini¬ 
exercises  for  practicing  the  steps,  and  two  exercises  for  performing  the  robotic  control  tasks  (one 
for  practicing  the  tele-operation  task  and  one  for  practicing  the  auto  control  tasks).  After  the 
tutorial  on  TCU,  participants  were  trained  in  the  gunnery  tasks.  Participants  had  to  demonstrate 
that  they  could  perform  the  tasks  without  any  help.  The  entire  training  session  lasted  about  2.5 
hours. 

Before  the  experimental  session  began,  participants  changed  into  one  of  the  laboratory  cotton 
T-shirts  in  order  to  standardize  how  the  tactors  were  applied  to  the  skin.  They  chose  a  size  and 
then  were  escorted  to  the  restroom  where  they  could  change  privately.  After  this  was  done,  the 
experimenter  asked  to  measure  the  participant  around  the  abdomen  just  above  navel  height  so 
that  the  tactile  display  could  be  custom  fitted.  After  taking  this  measurement,  the  experimenter 
arranged  the  tactors  so  that  they  were  equidistant  for  the  participant’s  abdomen.  Once  fitted 
with  the  tactile  display,  the  participant  was  seated  in  front  of  the  gunner  monitor.  A  test  pattern 
confirmed  that  all  eight  tactors  were  working  properly  and  that  the  participant  could  readily 
perceive  the  stimuli.  The  experimenter  then  explained  the  nature  of  the  AiTR  system  and  the 
corresponding  visual  or  tactile  cues  that  would  be  provided. 

In  the  experimental  trials,  participants’  tasks  were  to  use  their  robotic  asset  to  locate  targets  (i.e., 
enemy  dismounted  Soldiers)  in  the  remote  environment  and  to  find  targets  in  their  immediate  (i.e., 
MCS)  environment.  The  MCS  was  simulated  as  traveling  along  a  designated  route,  which  was 
approximately  4.3  km  and  lasted  about  15  minutes.  There  were  10  hostile  and  10  neutral  targets 
(i.e.,  civilians)  along  the  route  in  each  gunnery  scenario.  Participants  were  instructed  to  engage  the 
hostile  targets  and  verbally  report  spotting  the  neutral  targets.  In  total,  there  were  three  15-minute 
scenarios,  corresponding  to  the  three  within-subject  experimental  conditions,  the  order  of  which 
was  counterbalanced  across  participants. 

There  were  three  types  of  robotics  tasks:  Monitor,  Auto,  and  Teleop.  The  Monitor  task  required 
the  operator  to  continuously  monitor  the  video  feed  as  the  UGV  travels  autonomously  and  to 
verbally  report  detection  of  targets.  There  were  20  targets  (5  hostile  and  15  neutral)  along  the 
route.  The  Auto  control  task  required  the  operator  to  monitor  the  video  feed  as  the  UGV  traveled 
autonomously,  examine  still  images  generated  from  the  RSTA  (reconnaissance,  surveillance,  and 
target  acquisition)  scans,  and  detect  targets.  The  Teleop  task  required  the  operator  to  manually 
manipulate  and  drive  the  UGV  (using  a  joystick)  along  a  predetennined  route  using  the  TCU  to 
detect  targets.  For  both  the  Auto  and  Teleop  tasks,  there  were,  on  average,  two  to  four  targets  (one 

“PowerPoint  is  a  trademark  of  Microsoft  Corporation. 
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hostile  plus  one  to  three  neutral  targets)  at  each  checkpoint.  There  were  five  checkpoints  along  the 
designated  route.  Upon  detecting  a  target,  participants  needed  to  place  the  target  on  the  map,  label 
the  target,  and  then  send  a  spot  report. 

While  the  participants  were  perfonning  their  gunnery  and  robotics  control  tasks,  they  simultane¬ 
ously  performed  the  communication  task  by  answering  questions  delivered  to  them  via  DECtalk. 

There  were  2-minute  breaks  between  experimental  scenarios.  Participants  assessed  their  workload 
using  the  NASA  TLX  after  they  completed  each  scenario.  The  entire  experimental  session  lasted 
about  1  hour. 

2.5  Measures 

The  dependent  measures  include  mission  performance  (i.e.,  number  of  targets  detected  in  the 
remote  environment  with  the  robotic  asset  and  number  of  hostile/neutral  targets  detected  in  the 
immediate  environment),  communication  task  perfonnance,  and  perceived  workload. 


3.  Results 


3.1  Target  Detection  Performance 

3.1.1  Gunnery  Tasks 

3 . 1 . 1 . 1  Hostile  T argets 

Table  1  lists  several  measures  relating  to  operator  task  performance.  A  mixed  analysis  of  variance 
(ANOVA)  was  perfonned  to  examine  the  effects  of  the  concurrent  robotic  control  tasks  on  the 
gunnery  task  performance  (percentage  of  hostile  targets  detected),  with  the  AiTR  condition  (FAP 
versus  MP)  being  the  between-subject  factor  and  the  Robotics  Task  condition  (Monitor  versus  Auto 
versus  Teleop)  as  the  within-subject  factor  (see  appendix  E  for  the  complete  data).  The  analysis 
revealed  that  the  Robotics  condition  significantly  affected  number  of  targets  detected,  F( 2,  15)  = 
4.578  ,p<  .05  (figure  5).  Post  hoc  tests  (least  significant  difference  [LSD])  showed  that  target 
detection  in  the  Monitor  condition  was  significantly  higher  than  in  the  Auto  and  Teleop  conditions. 


Table  1.  Operator  task  performance  (mean  percent  of  targets  detected  for  the  gunnery  and  robotics  tasks  and 
percent  correct  for  the  communication  task.  (Standard  deviations  are  presented  in  parentheses.) 


Measures 

FAP 

MP 

Monitor 

Auto 

Teleop 

Monitor 

Auto 

Teleop 

Gunnery  Task 

.775 

.551 

.644 

.608 

.533 

.500 

(Hostile  only) 

(.238) 

(.302) 

(.256) 

(.156) 

(.227) 

(.268) 

Gunnery  Task 

.325 

.442 

.319 

.583 

.467 

.413 

(Neutral  only) 

(.166) 

(.178) 

(.171) 

(.255) 

(.267) 

(.226) 

Robotics  Task 

.908 

.639 

.770 

.906 

.742 

.637 

(.104) 

(.187) 

(.134) 

(.102) 

(.158) 

(.128) 

Communication 

.802 

.773 

.777 

.870 

.812 

.805 

Task 

(.118) 

(.112) 

(.133) 

(.089) 

(.117) 

(.100) 
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Figure  5.  Gunnery  task  performance,  hostile  targets  only. 

The  effects  of  participants’  SpA  and  PAC  were  also  evaluated.  The  analysis  showed  that  partici¬ 
pants  with  higher  SpA  had  a  significantly  higher  gunnery  task  performance  than  did  those  with 
lower  SpA,  F(  1,  16)  =  6.31 1,/?  <  .05  (figure  6). 

There  was  also  a  significant  AiTR  x  PAC  interaction,  F(  1 ,  16)  =  7.358,  p  <  .05  (figure  7).  Those 
with  a  lower  PAC  had  a  better  performance  with  the  FAP  cueing,  and  those  with  a  higher  PAC  had 
a  better  performance  with  the  MP  cueing. 

In  order  to  further  examine  the  effect  of  task  load  on  reliance  of  AiTR,  the  data  of  the  MP  condi¬ 
tion  were  analyzed  separately.  Because  of  the  small  sample  size  (N  =  12),  no  significant  differ¬ 
ences  were  found  between  those  with  a  high  versus  low  PAC.  However,  the  trend  was  evident  that 
as  task  load  became  heavier  (i.e.,  Teleop  >  Auto  >  Monitor,  based  on  Chen  &  Joyner,  2006),  those 
with  a  low  PAC  became  increasingly  reliant  on  the  AiTR,  while  those  with  a  high  PAC  maintained 
a  fairly  stable  level  of  reliance  throughout  the  experimental  conditions  (figure  8). 
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Figure  6.  Effects  of  spatial  ability  on  gunnery  task  performance,  hostile  targets  only. 


Figure  7.  Effects  of  perceived  attentional  control  on  gunnery  task  performance,  hostile  targets  only. 
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Figure  8.  Effects  of  perceived  attentional  control  on  gunnery  task  performance  (hostile 
targets  only)  in  MP  conditions. 

3 . 1 . 1 .2  Neutral  Targets 

Participants’  detection  of  neutral  targets  was  also  assessed.  Since  the  AiTR  only  alerted  the  par¬ 
ticipants  when  hostile  targets  were  present,  the  neutral  target  detection  could  be  used  to  indicate 
how  much  visual  attention  was  devoted  to  the  gunnery  station.  A  mixed  ANOVA  revealed  a 
significant  main  effect  for  Robotics,  F(2,15)  =  4.362,  p  <  .05  (figure  9).  Post  hoc  tests  (LSD) 
showed  that  neutral  target  detection  in  the  Teleop  condition  was  significantly  lower  than  in  the 
Auto  condition.  There  was  also  a  significant  AiTR  x  PAC  interaction,  F(l,  16)  =  3.602,  p  <  .05 
(figure  10).  Those  with  a  lower  PAC  performed  at  about  the  same  level,  regardless  of  the  AiTR 
type,  while  those  with  a  higher  PAC  had  a  significantly  better  performance  with  the  MP  cueing. 
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Figure  9.  Gunner’s  neutral  target  detection  performance. 


Figure  10.  Effects  of  perceived  attentional  control  on  gunner’s  neutral  target  detection 
performance. 
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3.1.2  Robotics  Tasks 


A  mixed  ANOVA  revealed  that  there  was  a  significant  main  effect  for  Robotics,  F(2,15)  =  25.357, 
p  <  .001  (figure  1 1).  It  was  found  that  the  Monitor  condition  was  significantly  higher  than  both  the 
Auto  and  the  Teleop  conditions  in  terms  of  percentage  of  targets  detected.  There  was  a  significant 
Robotics  x  AiTR  interaction,  F( 2,32)  =  3.96,  p  <  .05.  The  Monitor  task  performance  stayed  at  the 
same  level,  regardless  of  the  AiTR  types.  The  Auto  task  perfonnance  was  higher  with  the  MP 
cueing,  while  the  Teleop  task  performance  was  higher  with  the  FAP  cueing. 


Figure  11.  Robotics  task  performance. 

There  was  also  a  significant  AiTR  x  PAC  interaction,  TfiTJO)  =  4.801,/?  <  .05  (figure  12).  Those 
with  a  lower  PAC  had  a  better  performance  with  the  FAP  cueing,  while  those  with  a  higher  PAC 
had  a  better  performance  with  the  MP  cueing. 
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Figure  12.  Effects  of  perceived  attentional  control  on  robotics  task  performance. 

3.2  Communication  Task  Performance 

A  mixed  ANOVA  revealed  that  there  was  a  significant  main  effect  for  Robotics,  ^(2,44)  =  3.247, 
p  <  .05  (figure  13).  It  was  found  that  the  Monitor  condition  was  significantly  higher  than  the 
Teleop  conditions. 

3.3  Perceived  Workload 

Weighted  ratings  of  the  scales  of  the  NASA  TLX  were  used  for  this  analysis.  Participants’  self- 
assessment  of  workload  was  significantly  affected  by  Robotic  condition,  F(2,15)  =  25.057, 
p  <  . 001  (figure  14).  The  perceived  workload  was  significantly  higher  in  the  Teleop  condition 
(M=  77.71)  than  in  the  Auto  condition  (M=  69.61)  and  the  Monitor  condition  (M=  61.06).  The 
difference  between  Auto  and  Monitor  was  also  significant.  There  was  a  significant  Robotics  x 
AiTR  interaction,  7^(2, 1 5)  =  5.464,  p  <  .05. 
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Figure  13.  Communication  task  performance. 


Figure  14.  Perceived  workload. 
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3.4  AiTR  Display  Assessment 

A  usability  questionnaire  captured  participant  preferences  for  the  presentation  of  AiTR  informa¬ 
tion.  Following  their  interaction  with  the  AiTR  systems,  41%  of  participants  responded  that  they 
relied  predominantly  or  entirely  on  the  tactile  AiTR  display,  while  36%  responded  that  they  relied 
predominantly  or  entirely  on  the  visual  AiTR  display.  AiTR  preference  was  also  significantly 
correlated  with  SpA  (r  =  .507,  p  <  .01).  Those  with  higher  SpA  tended  to  prefer  tactile  cueing  over 
visual  cueing.  Conversely,  those  with  lower  SpA  favored  visual  cueing  over  tactile  cueing. 

We  also  evaluated  participants’  self-assessed  trust  in  the  AiTR  system  and  compared  those  with 
higher  and  lower  PAC.  There  was  no  significant  difference  between  these  two  groups. 


4.  Discussion 


In  this  study,  we  simulated  a  generic  mounted  environment  and  conducted  an  experiment  to 
examine  the  performance  and  workload  of  the  combined  position  of  gunner  and  robotics  operator. 
More  specifically,  we  investigated  the  effects  of  AiTR  with  imperfect  reliability  (FAP  versus  MP) 
on  the  operator’s  perfonnance  of  the  automated  (i.e.,  gunnery)  task  as  well  as  the  concurrent  tasks 
(i.e.,  robotics  and  communication).  Our  most  intriguing  findings  were  related  to  the  relationship  of 
individual  differences  to  predicted  compliance  and  reliance  perfonnance.  In  order  to  simplify  our 
discussion,  we  discuss  effects  of  PAC  differences  on  gunner  performance  for  FAP  and  MP  con¬ 
ditions  and  then  discuss  how  these  differences  affected  neutral  targeting  and  secondary  task 
performance  (robotics  and  communication). 

4.1  Gunnery  Task  Performance,  Hostile  Targets 

Our  results  showed  that  the  operator’s  gunnery  task  perfonnance  in  detecting  hostile  targets  was 
significantly  better  in  the  Monitor  condition  than  in  the  other  two  robotics  task  conditions,  consis¬ 
tent  with  the  findings  of  Chen  and  Joyner  (2006).  In  Chen  and  Joyner  (2006)  and  the  current 
study,  the  workload  associated  with  the  Monitor  condition  was  significantly  lower  than  the  other 
robotics  conditions  (see  section  4.5  for  more  discussion  of  workload).  These  results  suggest  that 
the  operator  had  more  visual  and  mental  resources  for  the  gunnery  task  when  the  robotics  task  was 
simply  monitoring  the  video  feed,  compared  with  the  other  two  robotics  conditions. 

Also  consistent  with  the  previous  findings  (Chen  &  Joyner,  2006;  Chen  &  Terrence,  2007), 
participants’  SpA  was  found  to  be  an  accurate  predictor  of  their  gunnery  performance.  Thomas 
and  Wickens  (2004)  showed  that  there  were  individual  differences  in  scanning  effectiveness  and 
its  associated  target  detection  performance.  However,  Thomas  and  Wickens  did  not  examine  the 
characteristics  of  those  participants  who  had  more  effective  scanning  strategies.  The  findings  of 
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the  current  study  along  with  the  previous  two  studies  indicate  that  SpA  may  be  an  important  factor 
for  detennining  scanning  effectiveness. 


Our  results  also  showed  that  there  was  a  significant  interaction  between  types  of  unreliable  AiTR 
and  participants’  PAC.  For  those  with  high  PAC,  our  data  are  consistent  with  the  notion  that 
operator  reliance  on  and  compliance  with  automation  are  independent  constructs  and  are  separately 
affected  by  system  misses  and  false  alarms.  Based  on  figure  7,  it  is  evident  that  high  PAC  partici¬ 
pants  did  not  comply  with  alerts  in  the  FAP  condition.  Since  the  FAP  AiTR  had  a  0%  miss  rate,  a 
full  compliance  should  result  in  a  detection  rate  over  80%,  as  reported  in  Chen  and  Terrence  (in 
which  a  perfectly  reliable  AiTR  was  used).  As  predicted,  figure  7  shows  that  in  MP  conditions, 
high  PAC  participants  did  not  rely  on  the  AiTR  and  detected  more  targets  than  were  cued.  How¬ 
ever,  an  examination  of  the  data  for  the  low  PAC  participants  revealed  a  completely  opposite 
trend.  Specifically,  with  the  FAP  condition,  low  PAC  participants  showed  a  strong  compliance 
with  the  alerts,  which  resulted  in  a  good  perfonnance  in  target  detection  (at  a  similar  level  as  in 
Chen  and  Terrence).  With  the  MP  condition,  however,  low  PAC  participants  evidently  overly 
relied  on  the  automation  and  therefore  had  a  very  poor  perfonnance.  Figure  8  shows  that  as  task 
load  became  heavier,  those  with  low  PAC  became  increasingly  reliant  on  the  AiTR,  while  those 
with  high  PAC  maintained  a  fairly  stable  level  of  reliance  throughout  the  experimental  conditions. 
According  to  Biros,  Daly,  and  Gunsch  (2004),  higher  task  loads  tend  to  induce  a  higher  level  of 
reliance  on  automated  systems.  Our  present  data  suggest  that  this  heightened  level  of  reliance  is 
also  moderated  by  PAC.  More  specifically,  only  those  with  low  PAC  tend  to  exhibit  over-reliance 
on  automation  under  a  heavy  task  load.  Because  of  the  small  sample  size,  this  observation  could 
not  be  confirmed  based  on  statistical  tests.  Future  research  should  further  examine  this  relation¬ 
ship  between  PAC,  task  load,  and  reliance  on  automation. 

4.2  Gunnery  Task  Performance,  Neutral  Targets 

It  was  found  that  the  gunner’s  detection  of  neutral  targets  (which  was  not  aided  by  AiTR)  was 
significantly  worse  when  s/he  had  to  tele-operate  a  robotic  asset  (versus  when  the  asset  was  semi- 
autonomous).  This  finding  is  consistent  with  previous  research  (Chen  &  Joyner,  2006;  Chen  & 
Terrence,  2007)  and  suggests  that  participants  devoted  significantly  less  visual  attention  to  the 
gunnery  station  when  their  robotic  asset  required  tele-operation.  Our  data  also  showed  that  those 
with  a  lower  PAC  performed  at  about  the  same  level,  regardless  of  the  AiTR  type,  while  those 
with  a  higher  PAC  had  a  significantly  better  perfonnance  with  the  MP  cueing.  This  suggests  that 
higher  PAC  participants  devoted  more  visual  attention  to  the  gunnery  station  (implying  a  reduced 
reliance  on  automation  for  the  gunnery  task)  when  the  AiTR  was  MP  than  when  the  AiTR  was 
FAP.  Although  we  did  not  measure  participants’  scanning  behaviors,  the  detection  rate  of  neutral 
targets  on  the  gunnery  station  provides  an  estimate  of  the  amount  of  operator’s  visual  attention  on 
the  automated  task  environment.  Again,  the  data  of  high  PAC  participants  seem  to  support  the 
hypothesis  that  MP  automation  reduces  operator  reliance.  However,  the  same  phenomenon  was 
not  observed  for  the  low  PAC  participants. 
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4.3  Robotics  Task  Performance 

For  the  robotics  tasks,  the  data  showed  that  participants  had  the  best  perfonnance  when  the  task 
was  only  monitoring  the  video  feed.  Moreover,  the  Monitor  task  performance  stayed  at  the  same 
level,  regardless  of  the  AiTR  types.  On  the  other  hand,  the  Auto  task  performance  was  higher 
with  the  MP  cueing,  while  the  Teleop  task  perfonnance  was  higher  with  the  FAP  cueing.  The 
data  of  the  Teleop  performance  are  consistent  with  previous  studies  that  MP  automation  degrades 
concurrent  task  perfonnance  more  than  FAP  (Dixon  &  Wickens,  2006;  Wickens,  Dixon,  Goh  et 
ah,  2005).  However,  the  same  trend  was  not  observed  for  the  other  two  robotics  tasks,  which  were 
less  challenging  than  the  Teleop  task.  Therefore,  it  appears  that  the  adverse  effect  of  MP  automa¬ 
tion  on  concurrent  tasks  is  only  manifest  in  more  challenging  task  conditions.  Our  data  also 
showed  that  again,  there  was  a  significant  AiTR  Type  x  PAC  interaction.  Consistent  with  the 
previous  two  performance  measures  (gunnery-hostile  and  gunnery-neutral),  the  low  PAC  parti¬ 
cipants  exhibited  a  higher  level  of  performance  degradation  with  the  MP  conditions.  The  per¬ 
formance  of  the  high  PAC  participants,  on  the  other  hand,  showed  a  completely  opposite  trend. 
These  results  suggest  that  the  high  PAC  participants’  reduced  compliance  with  the  FAP  alerts  did 
not  help  them  with  their  concurrent  task,  compared  with  the  MP  conditions;  in  fact,  it  resulted  in  a 
poorer  targeting  performance.  Conversely,  their  reduced  reliance  on  the  MP  alerts  did  not  hurt 
their  performance  as  did  the  FAP  alerts.  Overall,  the  low  PAC  participants  showed  the  most  pro¬ 
nounced  adverse  effect  of  MP  alerts  on  concurrent  perfonnance.  Conversely,  the  FAP  alerts  not 
only  helped  them  with  their  automated  task  but  also  their  concurrent  task. 

Taking  the  three  main  performance  measures  together  (i.e.,  Gunnery  task,  Hostile  Targets,  Gunner 
task,  Neutral  Targets,  and  Robotics  task),  it  appears  that  overall,  for  high  PAC  participants,  FAP 
alerts  were  more  detrimental  than  MP  alerts.  FAP  alerts  affected  their  automated  task  and  the 
concurrent  task.  This  finding  is  consistent  with  the  conclusion  of  Dixon  et  al.  (2007)  that  FAP 
degraded  overall  perfonnance  more  than  MP  automation.  However,  it  is  worth  noting  that  for  low 
PAC  participants,  we  observed  the  opposite  pattern:  MP  automation  was  more  harmful  than  FAP 
automation.  The  overall  data  suggest  that  low  PAC  participants  had  a  higher  trust  in  the  automa¬ 
tion  system  than  did  high  PAC  participants.  It  is  likely  that  low  PAC  participants  had  more 
difficulty  in  performing  multiple  tasks  concurrently  and  had  to  rely  on  automation  when  available. 
High  PAC  participants,  on  the  other  hand,  tended  to  rely  on  their  own  multi-tasking  ability  to 
perform  the  tasks.  It  is  interesting  to  note  that  there  was  no  significant  difference  in  the  partici¬ 
pants’  self  assessment  of  their  trust  in  the  AiTR  system  between  high  PAC  and  low  PAC  groups. 
Our  results  are  consistent  with  past  research  (Lee  &  Moray,  1994;  Lee  &  See,  2004)  that  self- 
confidence  is  a  critical  factor  in  mediating  the  effect  of  trust  (in  automation)  on  reliance  (on  the 
automatic  system).  Lee  and  Moray  found  that  when  self-confidence  exceeded  trust,  operators 
tended  to  use  manual  control.  When  trust  exceeded  self-confidence,  automation  was  used  more. 
Further  research  should  be  conducted  to  examine  the  relationships  between  PAC,  self-confidence, 
and  trust  in  automation. 
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4.4  Communication  Task  Performance 


Participants’  communication  task  performance  was  significantly  better  when  their  robotics  task 
was  Monitor  than  when  it  was  Teleop.  This  finding  is  consistent  with  Chen  and  Joyner  (2006). 
According  to  Naveh-Benjamin,  Craik,  Perretta,  and  Tonev  (2000),  infonnation-encoding  processes 
require  more  attention  than  retrieval  and  are  more  prone  to  the  effects  of  competing  demands  of 
multi-tasking.  It  is  therefore  likely  that  the  information-encoding  process  of  the  communication 
task  in  our  study  was  more  disrupted  by  the  more  challenging  Teleop  task  than  by  the  Monitor 
task. 


4.5  Perceived  Workload 

Participants’  perceived  workload  was  found  to  be  affected  by  the  type  of  concurrent  robotics  task. 
The  workload  was  significantly  higher  in  the  Teleop  condition  than  in  the  Auto  condition,  which 
in  turn  was  significantly  higher  than  in  the  Monitor  condition.  These  results  are  consistent  with 
Chen  and  Joyner  (2006)  and  Schipani  (2003),  which  evaluated  robotic  operator  workload  in  a  field 
setting.  Although  many  of  the  ground  robotic  assets  in  the  Army’s  FCS  program  will  be  semi- 
autonomous,  tele-operation  will  still  be  an  important  part  of  any  missions  involving  robotic  assets 
(e.g.,  when  robots  encounter  obstacles  or  other  problems).  The  higher  workload  associated  with 
tele-operation  needs  to  be  taken  into  account  when  one  is  designing  the  user  interfaces  for  the 
robotic  assets  (see  Chen,  Haas,  &  Barnes,  2007,  for  a  review  of  user  interface  designs  for  tele- 
operated  robots). 

4.6  AiTR  Display  Preference 

The  significant  positive  correlation  of  AiTR  preference  with  SpA  is  interesting.  It  appeared  that  as 
AiTR  ratings  tended  toward  considerable  reliance  on  the  tactile  display,  there  was  a  concurrent 
shift  with  higher  performance  on  the  spatial  tests.  This  confirmed  a  trend  that  was  first  observed  in 
Chen  and  Terrence  (2007).  Perhaps  those  with  higher  SpA  can  more  easily  employ  the  spatial 
tactile  signals  in  the  dual  task  setting  and  therefore  have  a  stronger  preference  for  something  that 
makes  the  gunner  task  easier  to  complete.  Individuals  with  lower  SpA,  on  the  other  hand,  may 
have  not  used  the  spatial  tactile  cues  to  their  full  extent  and  therefore  continued  to  prefer  the  visual 
AiTR  display.  According  to  Kozhevnikov  Hegarty,  and  Mayer  (2002),  visualizers  with  lower  SpA 
tend  to  rely  on  iconic  imagery  while  those  with  higher  SpA  tend  to  prefer  using  spatial-schematic 
imagery  to  solve  problems.  Therefore,  it  is  likely  that  in  our  study,  those  who  preferred  visual 
AiTR  displays  might  be  more  iconic  in  their  mental  representations.  However,  this  preference 
may  have  caused  degraded  target  detection  performance  because  more  visual  attention  was 
devoted  to  the  visual  AiTR  display  than  to  the  simulated  environment.  In  contrast,  those  who 
were  more  spatial  relied  on  the  directional  infonnation  of  the  tactile  display  to  help  them  with  the 
visually  demanding  tasks,  resulting  in  a  more  effective  performance. 
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5.  Conclusions 


In  this  study,  we  simulated  a  generic  mounted  environment  and  conducted  an  experiment  to 
examine  the  performance  and  workload  of  the  combined  position  of  gunner  and  robotics  operator. 
More  specifically,  we  investigated  the  effects  of  AiTR  with  imperfect  reliability  (FAP  versus  MP) 
on  the  operator’s  perfonnance  of  the  automated  (i.e.,  gunnery)  task  as  well  as  the  concurrent  tasks 
(i.e.,  robotics  and  communication).  Our  data  suggest  that  there  is  a  strong  interaction  between  the 
type  of  AiTR  unreliability  and  participants’  PAC  for  almost  all  the  performance  measures  (except 
for  the  communication  task,  which  showed  the  same  trend  but  the  interaction  was  not  statistically 
significant).  Overall,  it  appears  that  for  high  PAC  participants,  FAP  alerts  were  more  detrimental 
than  MP  alerts.  FAP  alerts  affected  not  only  their  automated  task  but  also  the  concurrent  task. 
However,  for  low  PAC  participants,  MP  automation  was  more  harmful  than  FAP  automation. 
Future  research  should  incorporate  performance-based  measures  of  attentional  shifting  effective¬ 
ness  (e.g.,  Synthetic  Work  Environment  or  SYNWORK)  in  addition  to  surveys  such  as  the  ACS 
(Branscome  &  Grynovicki,  2007).  In  the  area  of  SpA,  the  current  study  replicated  the  finding  of 
Chen  and  Terrence  (2007)  that  the  operator’s  preference  of  AiTR  display  (i.e.,  visual  versus 
tactile)  is  correlated  with  his  or  her  SpA.  Low  SpA  individuals  prefer  visual  cueing  over  tactile 
cueing,  although  tactile  display  would  be  more  effective  in  highly  visual  environments  (so  visual 
attention  can  be  devoted  to  the  tasks,  not  to  the  cues).  These  findings  may  have  important 
implications  for  personnel  selection,  system  designs,  and  training  development.  For  example,  to 
better  enhance  the  task  perfonnance  for  low  SpA  individuals,  the  visual  cueing  display  should  be 
more  integrated  with  the  visual  scene.  Augmented  reality  (i.e.,  visual  overlays)  is  a  potential 
technique  to  embed  directional  information  onto  the  video  (Calhoun  &  Draper,  2006).  Addition¬ 
ally,  the  capabilities  and  limits  of  the  automated  systems  should  be  conveyed  to  the  operator,  when 
feasible,  in  order  for  the  operator  to  develop  appropriate  trust  and  reliance  (Lee  &  See,  2004). 
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Appendix  A.  Demographic  Questionnaire 


Participant# _  Age _  Major _  Date _  Gender _ 

1.  What  is  the  highest  level  of  education  you  have  had? 

Less  than  4  yrs  of  college _  Completed  4  yrs  of  college _  Other _ 

2.  When  did  you  use  computers  in  your  education?  ( Circle  all  that  apply ) 

Grade  School  Jr.  High  High  School 

Technical  School  College  Did  Not  Use 

3.  Where  do  you  currently  use  a  computer?  (Circle  all  that  apply 1 

Home  Work  Library  Other _  Do  Not  Use 

4.  For  each  of  the  following  questions,  circle  the  response  that  best  describes  you. 

How  often  do  you: 

Use  a  mouse?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Use  a  joystick?  Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

Use  a  touch  screen?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Use  icon-based  programs/software? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 
Use  programs/software  with  pull-down  menus? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 
Use  graphics/drawing  features  in  software  packages? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 
Use  E-mail?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Operate  a  radio  controlled  vehicle  (car,  boat,  or  plane)? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

Play  computer/video  games? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

5.  Which  type(s)  of  computer/video  games  do  you  most  often  play  if  you  play  at  least  once  every  few  months? 

6.  Which  of  the  following  best  describes  your  expertise  with  computer?  (check  V  one) 

_ Novice 

_ Good  with  one  type  of  software  package  (such  as  word  processing  or  slides) 

_ Good  with  several  software  packages 

_ Can  program  in  one  language  and  use  several  software  packages 

_ Can  program  in  several  languages  and  use  several  software  packages 

7.  Are  you  in  your  usual  state  of  health  physically?  YES  NO 
If  NO,  please  briefly  explain: 

8.  How  many  hours  of  sleep  did  you  get  last  night? _ hours 

9.  Do  you  have  normal  color  vision?  YES  NO 

10.  Do  you  have  prior  military  service?  YES  NO  If  Yes,  how  long _ 
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Appendix  B.  Attentional  Control  Survey 


For  each  of  the  following  questions,  circle  the  response  that  best  describes  you. 

It  is  very  hard  for  me  to  concentrate  on  a  difficult  task  when  there  are  noises  around. 

Almost  never,  Sometimes,  Often,  Always 

When  I  need  to  concentrate  and  solve  a  problem,  I  have  trouble  focusing  my  attention. 

Almost  never,  Sometimes,  Often,  Always 


When  I  am  working  hard  on  something,  I  still  get  distracted  by  events  around  me. 

Almost  never,  Sometimes,  Often,  Always 

My  concentration  is  good  even  if  there  is  music  in  the  room  around  me. 

Almost  never,  Sometimes,  Often,  Always 


When  concentrating,  I  can  focus  my  attention  so  that  I  become  unaware  of  what’s  going  on  in  the  room  around 
me.  Almost  never,  Sometimes,  Often,  Always 

When  I  am  reading  or  studying,  I  am  easily  distracted  if  there  are  people  talking  in  the  same  room. 

Almost  never,  Sometimes,  Often,  Always 


When  trying  to  focus  my  attention  on  something,  I  have  difficulty  blocking  out  distracting  thoughts. 

Almost  never,  Sometimes,  Often,  Always 

1  have  a  hard  time  concentrating  when  I’m  excited  about  something. 

Almost  never,  Sometimes,  Often,  Always 


When  concentrating,  I  ignore  feelings  of  hunger  or  thirst.  Almost  never,  Sometimes,  Often,  Always 


I  can  quickly  switch  from  one  task  to  another.  Almost  never,  Sometimes,  Often,  Always 

It  takes  me  a  while  to  get  really  involved  in  a  new  task.  Almost  never,  Sometimes,  Often,  Always 

It  is  difficult  for  me  to  coordinate  my  attention  between  the  listening  and  writing  required  when  taking  notes 
during  lectures.  Almost  never,  Sometimes,  Often,  Always 

I  can  become  interested  in  a  new  topic  very  quickly  when  I  need  to. 

Almost  never,  Sometimes,  Often,  Always 

It  is  easy  for  me  to  read  or  write  while  I’m  also  talking  on  the  phone. 

Almost  never,  Sometimes,  Often,  Always 

I  have  trouble  carrying  on  two  conversations  at  once.  Almost  never,  Sometimes,  Often,  Always 


I  have  a  hard  time  coming  up  with  new  ideas  quickly.  Almost  never,  Sometimes,  Often,  Always 

After  being  interrupted  or  distracted,  I  can  easily  shift  my  attention  back  to  what  I  was  doing  before. 

Almost  never,  Sometimes,  Often,  Always 

When  a  distracting  thought  comes  to  mind,  it  is  easy  for  me  to  shift  my  attention  away  from  it. 

Almost  never,  Sometimes,  Often,  Always 


It  is  easy  for  me  to  alternate  between  two  different  tasks.  Almost  never,  Sometimes,  Often,  Always 

It  is  hard  for  me  to  break  from  one  way  of  thinking  about  something  and  look  at  it  from  another  point  of  view. 

Almost  never,  Sometimes,  Often,  Always 
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Appendix  C.  NASA  TLX  Questionnaire 


Please  rate  your  overall  impression  of  demands  imposed  on  you  during  the  exercise. 

1.  Mental  Demand:  How  much  mental  and  perceptual  activity  was  required  (e.g.,  thinking,  looking, 
searching,  etc.)?  Was  the  task  easy  or  demanding,  simple  or  complex,  exacting  or  forgiving? 

LOW  |— |— |— |— |— |— I— I— I— I  HIGH 
123456789  10 

2.  Physical  Demand:  How  much  physical  activity  was  required  (e.g.,  pushing,  pulling,  turning, 
controlling,  activating,  etc.)?  Was  the  task  easy  or  demanding,  slow  or  brisk,  slack  or  strenuous,  restful  or 
laborious? 


LOW  l-l-l-l-l-l-l-l-l-l  HIGH 
123456789  10 

3.  Temporal  Demand:  How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace  at  which  the  task  or 
task  elements  occurred?  Was  the  pace  slow  and  leisurely  or  rapid  and  frantic? 

LOW  l-l-l-l-l-l-l-l-l-l  HIGH 

123456789  10 

4.  Level  of  Effort:  How  hard  did  you  have  to  work  (mentally  and  physically)  to  accomplish  your  level  of 
performance? 


LOW  l-l-l-l-l-l-l-l-l-l  HIGH 
123456789  10 

5.  Level  of  Frustration:  How  insecure,  discouraged,  irritated,  stressed  and  annoyed  versus  secure, 
gratified,  content,  relaxed  and  complacent  did  you  feel  during  the  task? 

LOW  l-l-l-l-l-l-l-l-l-l  HIGH 
123456789  10 

6.  Performance:  How  successful  do  you  think  you  were  in  accomplishing  the  goals  of  the  task  set  by  the 
experimenter  (or  yourself)?  How  satisfied  were  you  with  your  performance  in  accomplishing  these  goals? 

LOW  l-l-l-l-l-l-l-l-l-l  HIGH 

123456789  10 
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Appendix  D.  Usability  Questionnaire 


1.  The  gunner  station  should  only  have  the  visual  ATR  display. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

2.  The  gunner  station  should  only  have  the  tactile  ATR  display. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

3.  I  made  use  of  both  the  visual  and  tactile  ATR  displays. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

4.  I  sometimes  felt  ‘lost’  using  the  tactile  ATR  display. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

5.  I  sometimes  felt  ‘lost’  using  the  visual+tactile  display. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

6.  The  tactile  ATR  display  was  intuitive  and  made  it  easy  to 
determine  the  direction  of  targets. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

7.  The  visual+tactile  ATR  display  was  intuitive  and  made  it  easy  to 
determine  the  direction  of  targets. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 
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8.  I  do  not  feel  either  ATR  display  (tactile  or  visual+tactile)  was 
helpful  in  the  gunner  task. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

9.  The  tactile  ATR  display  was  helpful  when  I  had  to  teleoperate  the 
UGV. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

10.  I  relied  heavily  on  the  tactile  ATR  display  for  the  gunner  task. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

11.  I  relied  heavily  on  the  visual+tactile  ATR  display  for  the  gunner 
task. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

12.  The  tactile  ATR  display  was  helpful  when  the  UGV  was  semi- 
autonomous. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

13.  The  visual+tactile  ATR  display  was  helpful  when  I  had  to 
teleoperate  the  UGV. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

14.  The  visual+tactile  ATR  display  was  helpful  when  the  UGV  was 
semi-autonomous. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

15.  The  gunner  station  should  not  have  an  ATR  display. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

16.  The  tactile  ATR  display  was  confusing. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

17.  The  visual+tactile  ATR  display  was  confusing. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

18.  The  tactile  ATR  display  was  annoying. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

19.  The  visual+tactile  ATR  display  was  annoying. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 
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20.  The  tactile  ATR  display  improved  my  performance  on  the  gunner 
task. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

21.  The  visual+tactile  ATR  display  improved  my  performance  on  the 
gunner  task. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

22.  The  visual+tactile  ATR  display  can  be  deceptive. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

23.  The  visual+tactile  ATR  display  sometimes  behaves  in  an 
unpredictable  manner. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

24. 1  am  often  suspicious  of  the  visual+tactile  ATR  system’s  intent, 
action,  or  outputs. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

25. 1  am  sometimes  unsure  of  the  visual+tactile  ATR  system. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

26.  The  visual+tactile  ATR  system  may  have  harmful  effects  on  the 
gunnery  task. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

27. 1  am  confident  in  the  visual+tactile  ATR  system. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 
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28.  The  visual+tactile  ATR  system  can  provide  security. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

29.  The  visual+tactile  ATR  system  has  integrity. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

30.  The  visual+tactile  ATR  system  is  dependable. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

31.  The  visual+tactile  ATR  system  is  consistent. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

32.  I  can  trust  the  visual+tactile  ATR  system. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

33.  I  am  familiar  with  the  visual+tactile  ATR  display. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

34.  The  visual+tactile  ATR  display  improved  my  performance  on  the 
gunner  task. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


Which  of  the  following  best  describes  your  source  of  AITR  information  when  you  had  access  to 
both  the  visual  and  the  tactile  displays  (please  circle  ONE  answer  only): 

1 .  entirely  visual 

2.  predominately  visual 

3.  both  visual  and  tactile 

4.  predominately  tactile 

5.  entirely  tactile 
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Appendix  E.  Complete  Gunnery  Performance  Data  (hostile  targets) 


Gunnery  Task  (Hostile 
Targets)-  Auto  Condition 


Gunnery  Task  (Hostile 


AiTR 

SpA 

PAC 

Mean 

FAP 

Low 

Low 

.5750 

High 

.2500 

Total 

.4667 

High 

Low 

1.0000 

High 

.5622 

Total 

.6352 

Total 

Low 

.6600 

High 

.4730 

Total 

.5509 

MP 

Low 

Low 

.6500 

High 

.5500 

Total 

.5833 

High 

Low 

.4200 

High 

.8000 

Total 

.4833 

Total 

Low 

.4857 

High 

.6000 

Total 

.5333 

Total 

Low 

Low 

.6000 

High 

.4500 

Total 

.5250 

High 

Low 

.5167 

High 

.6018 

Total 

.5592 

Total 

Low 

.5583 

High 

.5259 

Total 

.5421 

FAP 

Low 

Low 

.6444 

Std.  Deviation 


.09574 
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High 


.3750 


.17678 


2 


Gunnery  Task  (Hostile 
Targets)-  Monitor  Condition- 


High 


Total 


MP  Low 


High 


Total 


Total  Low 


High 


Total 


FAP  Low 


High 


Total 

.5546 

Low 

1.0000 

High 

.6800 

Total 

.7333 

Low 

.7156 

High 

.5929 

Total 

.6440 

Low 

.3000 

High 

.5750 

Total 

.4833 

Low 

.4400 

High 

.9000 

Total 

.5167 

Low 

.4000 

High 

.6400 

Total 

.5000 

Low 

.5296 

High 

.5083 

Total 

.5190 

Low 

.5333 

High 

.7167 

Total 

.6250 

Low 

.5315 

High 

.6125 

Total 

.5720 

Low 

.7750 

High 

.5000 

Total 

.6833 

Low 

1.0000 

2 

.8400 

Total 

.8667 

.21529 

.27749 

.28048 

.22630 

.28052 

.25600 

.28284 

.33040 

.31885 

.16066 


.23647 

.18764 

.32094 

.26820 

.26149 

.28708 

.26204 

.27003 

.26394 

.27199 

.25343 

.28455 

.26675 

.32016 

.00000 

.28577 

.15166 

.15055 
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Total 

Low 

.8200 

.29496 

5 

High 

.7429 

.20702 

7 

Total 

.7750 

.23789 

12 

MP 

Low 

Low 

.5000 

.14142 

2 

High 

.5750 

.17078 

4 

Total 

.5500 

.15166 

6 

High 

Low 

.6400 

.15166 

5 

High 

.8000 

1 

Total 

.6667 

.15055 

6 

Total 

Low 

.6000 

.15275 

7 

High 

.6200 

.17889 

5 

Total 

.6083 

.15643 

12 

Total 

Low 

Low 

.6833 

.29269 

6 

High 

.5500 

.13784 

6 

Total 

.6167 

.22896 

12 

High 

Low 

.7000 

.20000 

6 

High 

.8333 

.13663 

6 

Total 

.7667 

.17753 

12 

Total 

Low 

.6917 

.23916 

12 

High 

.6917 

.19752 

12 

Total 

.6917 

.21451 

24 
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Acronyms 


ANOVA 

analysis  of  variance 

ARL 

Army  Research  Laboratory 

AiTR 

aided  target  recognition 

CR 

correct  rejection 

FA 

false  alarm 

FAP 

false  alarm  prone 

FCS 

Future  Combat  System 

FOV 

field  of  view 

IMPRINT 

Improved  Performance  Research  Integration  Tool 

LOS 

line  of  sight 

LSD 

least  significant  difference 

MCS 

mounted  combat  system 

MP 

miss  prone 

NASA  TLX 

National  Aeronautics  and  Space  Administration  Task  Load  Index 

ocu 

operator  control  unit 

OTW 

out  the  window 

PAC 

perceived  attentional  control 

RCTA 

Robotics  Collaborative  Technology  Alliance 

RSTA 

reconnaissance,  surveillance,  and  target  acquisition 

SIL 

system  integration  laboratory 

SpA 

spatial  ability 

SSQ 

Simulator  Sickness  Questionnaire 

TCU 

tactical  control  unit 

TS 

total  severity 
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TSS 

Total  Severity  Score 

UCD 

unmanned  combat  demonstration 

UCF 

University  of  Central  Florida 

UGV 

unmanned  ground  vehicle 

XUV 

experimental  unmanned  vehicle 
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1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
IMNE  ALC  IMS 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
AMSRD  ARL  Cl  OK  TL 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
AMSRD  ARL  CS  OK  T 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  ML  J  MARTIN 
MYER  CENTER  RM  2D3 1 1 
FT  MONMOUTH  NJ  07703-5601 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MZ  A  DAVISON 
320  MANSCEN  LOOP  STE  115 
FT  LEONARD  WOOD  MO  65473 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MD  T  COOK 
BLDG  5400  RM  C242 
REDSTONE  ARSENAL  AL  35898-7290 

1  COMMANDANT  USAADASCH 
ATTN  ATSACD 

ATTN  AMSRD  ARL  HR  ME  DR  HAWLEY 
5800  CARTER  RD 
FT  BLISS  TX  79916-3802 


1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MM  DR  V  J  RICE 
BLDG  4011  RM  217 
1750  GREELEY  RD 
FT  SAM  HOUSTON  TX  78234-5002 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MG  R  SPINE 
BUILDING  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARL  HRED  ARMC  FLD  ELMT 

ATTN  AMSRD  ARL  HR  MH  C  BURNS 
BLDG  1467B  ROOM  336 
THIRD  AVENUE 
FT  KNOX  KY  40121 

1  ARMY  RSCH  LABORATORY  -  HRED 
AWC  FIELD  ELEMENT 
ATTN  AMSRD  ARL  HR  MJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MK  MR  J  REINHART 
10125  KINGMAN  RD 
FT  BELVOIR  VA  22060-5828 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MV  HQ  USAOTC 
S  MIDDLEBROOKS 
91012  STATION  AVE  ROOM  348 
FT  HOOD  TX  76544-5073 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MY  M  BARNES 
2520  HEALY  AVE  STE  1 172  BLDG  51005 
FT  HUACHUCA  AZ  85613-7069 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MP  D  UNGV ARSKY 
POPE  HALL  BLDG  470  BCBL 
806  HARRISON  DR 
FT  LEAVENWORTH  KS  66027-2302 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MJF  J  HANSBERGER 
JFCOM  JOINT  EXPERIMENTATION  J9 
JOINT  FUTURES  LAB 
115  LAKEVIEW  PARKWAY  SUITE  B 
SUFFOLK  VA  23435 
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1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MQ  M  R  FLETCHER 
US  ARMY  SBCCOM  NATICK  SOLDIER  CTR 
AMSRD  NSC  WS  E  BLDG  3  RM  343 
NATICK  MA  01760-5020 

1 0  ARMY  RSCH  LABORATORY-HRED 
ATTN  AMSRD  ARL  HR  MT  J  CHEN 
12423  RESEARCH  PARKWAY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY-HRED 

ATTN  AMSRD  ARL  HR  MT  C  KORTENHAUS 
12350  RESEARCH  PARKWAY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MS  C  MANASCO 
SIGNAL  TOWERS  BLDG  29808A  RM  303 
FORT  GORDON  GA  30905-5233 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MU  M  SINGAPORE 
6501  E  11  MILE  RD  MAIL  STOP  284 
BLDG  200A  2ND  FL  RM  2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MF  C  HERNANDEZ 
2421  NW  AUSTIN  RD  STE  220 
FORT  SILL  OK  73503-9042 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MW  E  REDDEN 
BLDG  4  ROOM  332 
FT  BENNING  GA  31905-5400 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MN  R  SPENCER 
DCSFDI  HF 

HQ  USASOC  BLDG  E2929 
FORT  BRAGG  NC  28310-5000 

1  ARMY  G1 

ATTN  DAPE  MR  B  KNAPP 

300  ARMY  PENTAGON  ROOM  2C489 

WASHINGTON  DC  20310-0300 

1  ARL-HRED  LIAISON 

PHYSICAL  SCIENCES  LAB 
PO  BOX  30002 

LAS  CRUCES  NM  88003-8002 


1  DIRECTOR 

UNIT  OF  ACTION  MANEUVER  BATTLE  LAB 
ATTN  ATZKUA 
BLDG  1101 

FORT  KNOX  KY  40121 

1  DIR  FOR  PERS  TECHNOLOGIES 

DPY  CHIEF  OF  STAFF  PERS 
300  ARMY  PENTAGON  2C733 
WASHINGTON  DC  20310-0300 

1  CODE  1142PS 

OFC  OF  NAVAL  RSCH 
800  N  QUINCY  STREET 
ARLINGTON  VA  22217-5000 

1  CDR 

USA  AEROMEDICAL  RSCH  LAB 

ATTN  LIBRARY 

FORT  RUCKER  AL  36362-5292 

1  US  ARMY  NATICK  RD&E  CTR 
ATTN  STRNCYBA 
NATICK  MA  01760-5020 

1  PEO  STRI 

12350  RSCH  PARKWAY 
ORLANDO  FL  32826-3276 

1  GOVT  PUBLICATIONS  LIB 
409  WILSON  M 

UNIVERSITY  OF  MINNESOTA 
MINNEAPOLIS  MN  55455 

1  HUMAN  FACTORS  ENG  PROGRAM 
DEPT  OF  BIOMEDICAL  ENGNG 
COLLEGE  OF  ENG  &  COMPUTER  SCIENCE 
WRIGHT  STATE  UNIVERSITY 
DAYTON  OH  45435 

1  DIR  AMC-FIELD  ASSIST  IN 
SCIENCE  &  TECHNOLOGY 
ATTN  AMC-FAST 
FT  BELVOIR  VA  22060-5606 

ABERDEEN  PROVING  GROUND 

1  DIRECTOR 

US  ARMY  RSCH  LABORATORY 
ATTN  AMSRD  ARL  Cl  OK  (TECH  LIB) 

BLDG  4600 
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1  DIRECTOR 

US  ARMY  RSCH  LABORATORY 
ATTN  AMSRD  ARL  Cl  OK  S  FOPPIANO 
BLDG  459 

1  DIRECTOR 

US  ARMY  RSCH  LABORATORY 

ATTN  AMSRD  ARL  HR  MR  F  PARAGALLO 

BLDG  459 
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